Image quality

ABSTRACT

A magnified image is improved by integrating the wavelength specific component into that image. A magnified images obtained, and at least one wavelength specific component images also obtained. The different images are converted in color space, and different channels, indicative of the different parts of the image shows, are also obtained. For example, the image may be converted to and L*a*b* color space, and the luminance channel of the wavelength specific component may be used to enhance or replace the luminance channel of the magnified image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 60/716,887, filed on Sep. 13, 2005. The disclosure of the priorapplication is considered part of (and is incorporated by reference in)the disclosure of this application.

BACKGROUND

Pathology often requires viewing microscope images. The resolution ofthe microscope images from the imaging system. This is often limited bydifferent parameters of obtaining the image. For example, the resolutionmay be limited by the time it takes to scan a tissue section and by theresulting image file size.

DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flowchart;

FIG. 2 illustrates the progression of the different image;

FIG. 3 illustrates an exemplary hardware setup which can be used;

FIG. 4 a and 4 b show examples of the different images for colon cancer;and

FIG. 5 a through 5 d show examples of the different images for a breastcancer cell.

DETAILED DESCRIPTION

The general structure and techniques, and more specific embodimentswhich can be used to effect different ways of carrying out the moregeneral goals, are described herein.

The number of image elements within an obtained image from a tissuesection increases exponentially between different microscope objectives.For example, the image at 10× may require exponentially more storagethan the image at 4×. The time that is required to scan the tissuesection at 60× may be excessive. Therefore, many believe that capturinga large image at 60× is not practical. The time required to scan atissue section at 60× is extremely large, and the amount of digitalstorage space required for such a scan is also large. This may limit thenumber of scans that can be obtained and reviewed.

An embodiment describes use of a multi spectral imaging system, such asthe Nuance Multispectral Imaging System available from CambridgeResearch & Instrumentation (“Nuance”) in combination with an automatedmicroscope such as the Automated Cellular Imaging System (“ACIS”)provided by Clarient Inc. The processing provides an effectiveaugmentation of images at lower magnifications, to attempt to obtainadditional information from those images at lower magnifications. In anembodiment, image augmentation is carried out by extracting images atspecific color wavelengths, converting color spaces, and carrying outchannel mixing in a converted color space.

FIG. 3 illustrates an exemplary hardware setup which can be used. Thesample 300 is on a sample table 305 as conventional. The ACIS or otherautomated microscope 310 obtains image information from the sample 300.A single spectrum camera 315 also obtains information. All of theinformation is coupled to a computer 320 which operates as describedherein and specifically according to the flowcharts of FIGS. 1 and 2.

A color space is a model for representing color in terms of intensityvalues; a color space specifies how color information is represented. Itdefines a one, two, three, or four-dimensional space whose dimensions,or components, represent intensity values. A color component is alsoreferred to as a color channel. For example, RGB space is athree-dimensional color space whose components are the red, green, andblue intensities that make up a given color. Visually, these spaces areoften represented by various solid shapes, such as cubes, cones, orpolyhedral.

Different kinds of color spaces are known.

Gray spaces typically have a single component, ranging from black towhite. The RGB space is a three-dimensional color space whose componentsare the red, green, and blue intensities that make up a given color. Forexample, scanners read the amounts of red, green, and blue light thatare reflected from an image and then convert those amounts into digitalvalues. Displays receive the digital values and convert them into red,green, and blue light seen on a screen.

RGB-based color spaces are the most commonly used color spaces incomputer graphics, primarily because they are directly supported by mostcolor displays and scanners. RGB color spaces are device dependent andadditive. The groups of color spaces within the RGB base family includeHSV (hue, saturation, value) and HLS (hue, lightness, saturation)spaces. The saturation component in both color spaces describes colorintensity. A saturation value of 0 (in the middle of a hexagon) meansthat the color is “colorless” (gray); a saturation value at the maximum(at the outer edge of a hexagon) means that the color is at maximum“colorfulness” for that hue angle and brightness. The value component(in HSV space) and the lightness component (in HLS space) describebrightness or luminance. In both color spaces, a value of 0 representsblack. In HSV space, a maximum value means that the color is at itsbrightest. In HLS space, a maximum value for lightness means that thecolor is white, regardless of the current values of the hue andsaturation components. The brightest, most intense color in HLS spaceoccurs at a lightness value of exactly half the maximum.

CMY color spaces are like the above, but define the colors additively.

Any color expressed in RGB space is some mixture of three primarycolors: red, green, and blue. Most RGB-based color spaces can bevisualized as a cube, with corners of black, the three primaries (red,green, and blue), the three secondaries (cyan, magenta, and yellow), andwhite.

Some color spaces can express color in a device-independent way. WhereasRGB colors vary with display and scanner characteristics, and CMYKcolors vary with printer, ink, and paper characteristics,device-independent colors are meant to be true representations of colorsas perceived by the human eye. These color representations, calleddevice-independent color spaces, result from work carried out in 1931 bythe Commission Internationale d'Eclairage (CIE), and for that reason arealso also called CIE-based color spaces.

In the L*a*b color space, the L*a*b* space consists of a luminosity ‘L*’or brightness layer, chromaticity layer ‘a*’, indicating where colorfalls along the red-green axis, and chromaticity layer ‘b*’ indicatingwhere the color falls along the blue-yellow axis.

An embodiment is described herein. The embodiment can be carried outautomatically using a robotic or computer-controlled system.Alternatively, some parts of the embodiment, such as the staining or theinput of data into machines, can be carried out manually.

At 100, the system obtains a number of different images, including afirst microscopic image, and at least one single spectrum image.Preferably, a plurality of different single spectrum images areobtained. FIG. 2 also illustrates the different images, including thecolor image 200 from the microscope or from the spectral camera, and asingle spectrum image 205 from the spectral camera.

Colon cancer tissue sections may be examined in this embodiment. Thesesections are in fixed paraffin and stained with HER2 stain. Amultispectral camera, which in the embodiment can be the Nuance camera,is used to examine the tissue sections. The Nuance camera is mounted onan ACIS microscope.

For purposes of the embodiment, color RGB images are obtained at anymagnification, e.g., 4×, 10×, 20× and/or 60×. Grayscale images of theexact same fields are also captured at near ultraviolet (420 nm) andnear infrared (720 nm). Physics dictates that resolution is inverselyproportional to wavelength. One would therefore predict that the 420 nmimage would have better resolution than any of the RGB channels of theoriginal color images.

The images obtained from the cameras are in an RGB based color space. At110, the images are converted into a device independent color spacewhich includes a luminance component. More specifically, in theembodiment, the devices are converted into the L*a*b* color space. FIG.2 shows the image 200 being converted into the new color space image210, and the image 205 being converted into the new color space image215. This color space conversion may use commercially available softwareor modules.

At 120, the channels of the new images are separated. In FIG. 2, image210 is divided into separated channels, the L* channel 220, the a*channel 221 and the b* channel 222. Similarly, the image 215 isconverted into its separate channels representing separate image parts,225, 226 and 227.

In the embodiment, only the luminance information from the singlespectrum image 205 is used. Accordingly, at 130, the luminance channelsfrom the image 200 are replaced by the luminance channel from theircorresponding 420 nm image 205. The channels are then premixed at 140 tocreate the image 240, and then are transformed back to another colorspace transformation at 150 such as RGB, HSI, or any other color spaceof a type that may facilitate the display.

According to the embodiment, it was found that the new image providedmore detailed than the original. In order to test the importance of the420 nm image, the same process was done using a 720 nm spectral image inplace of the 420 nm image. The resulting images were of poor quality.

Another embodiment tested immunohistochemical stained tissue. Thistissue test was a breast-cancer test tissue stained with Her2/neu, usingdiainobenzidine (“DAB”) secondary, and a hematoxylin counterstain.Surprisingly, this process increased the detail of the hematoxylinstained counterstain tissue but greatly reduced the information carriedby the stained cancer tissue, which became less interpretable.

The inventor believes that the brown DAB based secondary stain containsa great deal of red color. Therefore, the 720 nm process was appliedwith very good results. The DAB stained tissue showed an increase detailat ends the background of slightly decreased background detail.

Therefore, the different convergences between different kinds of colorare important. FIGS. 4 a and 4 b show examples of the different imagesfor colon cancer. FIGS. 5 a through 5 d show examples of the differentimages for a breast cancer cell.

The embodiment describes only two different single spectrum images, butother embodiments may use a different luminance convert step 130 whichuses a plurality of different single spectrum images, or some kind ofcombined single spectrum image which is combined by transforming andweighting the number of different images together.

The general structure and techniques, and more specific embodimentswhich can be used to effect different ways of carrying out the moregeneral goals are described herein.

Although only a few embodiments have been disclosed in detail above,other embodiments are possible and the inventor (s) intend these to beencompassed within this specification. The specification describesspecific examples to accomplish a more general goal that may beaccomplished in another way. This disclosure is intended to beexemplary, and the claims are intended to cover any modification oralternative which might be predictable to a person having ordinary skillin the art. For example, other stains and colors may be used. Othersingle spectrum images, or images that are multispectrum or narrowspectrum can also be used. Moreover, when specific values, such as 420nm, are given herein, those specific values are intended to be centervalues within a range of 10-20%, for example.

Also, the inventor intends that only those claims which use the words“means for” are intended to be interpreted under 35 USC 112, sixthparagraph. Moreover, no limitations from the specification are intendedto be read into any claims, unless those limitations are expresslyincluded in the claims. The computers described herein may be any kindof computer, either general purpose, or some specific purpose computersuch as a workstation. The computer may be a Pentium class computer,running Windows XP or Linux, or may be a Macintosh computer. Thecomputer may also be a handheld computer, such as a PDA, cellphone, orlaptop.

The programs may be written in C, or Java, Brew or any other programminglanguage. The programs may be resident on a storage medium, e.g.,magnetic or optical, e.g. the computer hard drive, a removable disk ormedia such as a memory stick or SD media, or other removable medium, theprograms may also be run over a network, for example, with a server orother machine sending signals to the local machine, which allows thelocal machine to carry out the operations described herein.

1. A method, comprising: obtaining a first image comprising a microscopeimage indicative of a sample; obtaining a second image indicative of thesame sample but which covers substantially only a single spectrum;dividing said second image into component parts indicative of the secondimage; and using at least one of said component parts to enhance saidfirst image.
 2. A method as in claim 1, wherein said information fromsaid second image is a luminance component from the second image whichis used to modify a luminance component in the first image.
 3. A methodas in claim 1, further comprising converting said microscope image insaid second image to a second color space, separating channels in saidseparate color space, and using at least one of said channels from saidsecond image to replace at least one of said channels in said microscopeimage.
 4. A method as in claim 3, wherein said second color space is adevice independent color space.
 5. A method as in claim 3, wherein saidsecond color space is an L*a*b* color space.
 6. A method as in claim 3,after said using, combining said channels to form a new image, andtransforming said new image to another color space.
 7. A method as inclaim 1, wherein said second image is at substantially 420 nm.
 8. Amethod as in claim 1, wherein said second image is at 720 nm.
 9. Adevice comprising: a computer, obtaining an image of a microscope, andobtaining another image having a relationship to said microscope, butover substantially only a single spectrum, said computer operating toprovide information from the another image into component partsindicative of the another image and to use information from at least oneof said component parts, but not all of said component parts to enhancesaid image from said microscope.
 10. A device as in claim 9, furthercomprising a microscope, producing said image.
 11. A device as in claim9, further comprising a single spectrum camera, producing at least oneoutput indicative of a single spectrum image.
 12. A device as in claim10, further comprising a multi-spectrum camera, which produces a numberof output images, each representative of a single spectrum image.
 13. Adevice as in claim 9, wherein said at least one component part is acomponent part indicative of luminance within the image.
 14. A device asin claim 9, wherein said computer further operates to convert saidmicroscope image and said another image, into a device independent colorspace.
 15. A device as in claim 14, wherein said second color space isan L*a*b* color space.
 16. A device as in claim 9, wherein said computeroperates to combine said component parts to form a new image, and totransform said new image to another color space.
 17. A method as inclaim 11, wherein said single spectrum image is at substantially 420 nm.18. A method as in claim 11, wherein said single spectrum image is at720 nm.
 19. A method, comprising: obtaining a first image from a firstcamera indicative of full color image of a magnified sample; obtaining asecond image indicative of a single spectrum image of the magnifiedsample; converting said first and second images into a deviceindependent color space; separating channels of the first and secondimages in said device independent color space; using at least one ofsaid channels of the second image to enhance a quality of said firstimage; and creating a new image based on said first image, and said atleast one of said channels.
 20. A method as in claim 19, wherein saidusing comprises using said one channel from said a second image toreplace a corresponding channel in said first image.